‘Science by consensus’ impedes scientific creativity and progress: A simple alternative to funding biomedical research

The very low success rates of grant applications to the National Institutes of Health (NIH) and the National Science Foundation (NSF) are highly detrimental to the progress of science and the careers of scientists. The peer review process that evaluates proposals has been claimed arbitrarily to be the best there is. This consensus system, however, has never been evaluated scientifically against an alternative. Here we delineate the 15 major problems with the peer review process. We challenge the Science Advisor to the President, and the leadership of NIH, NSF, the U.S. National Academy of Sciences and other funding agencies throughout the world to refute each of these criticisms. We call for the implementation of more equitable alternatives that will not constrain the progress of science. We propose a system that will fund at least 80,000 principal investigators, including young scientists, with about half the current NIH budget, seven-times as many as the current number of NIH “research project grants,” and that will forego the cumbersome, expensive, and counterproductive “peer” review stage. Further, we propose that the success of the two systems over 5–10 years be compared scientifically.


Introduction
The success rate for National Institutes of Health (NIH) grants is currently 20% (NIH Report, 2022), and the "payline" for research grant applications by experienced principal investigators is 11% (NIAID Funding News, 2023).The funding rate at the National Science Foundation (NSF) was 26% in 2021 (National Science Foundation, 2022).The Gates Foundation does not even release its grant success rate information; it may be between 1 to 2%.In 2009 and 2010, NIH received more than 20,000 applications for its Challenge Grants funded through the American Recovery and Reinvestment Act; the success rate was only 4% (NIH Report, 2011).The "successful" projects are those that have been deemed by the consensus of "peers" to be worthwhile pursuing; hence, our designation for the way science appears to be carried out: "science by consensus."Even the designation, "peer review" is suspect, because review committees comprise many scientists who do not adequately know the field of the grant application.Despite these very low success rates and paylines that afflict the careers of the great majority of scientists, the peer review system has been claimed to be the best system there is to allocate funding for biomedical research (see, for example, Racker, 1979).This consensus system, however, has never been evaluated scientifically against an alternative (Düzgünes ¸, 1999).Thus, the reputation of the current system is not based on systematic comparisons against other unbiased and more rational systems.
Perhaps the earliest challenge to this system at NIH was made by John McGowan (1992), who was at the time the Director of Extramural Research at the National Institute of Allergy and Infectious Diseases (NIAID).He revealed that proposals to investigate human immunodeficiency virus (HIV) infections of macrophages had been rejected by a study section because "the literature does not support the hypothesis that HIV can grow in macrophages" (McGowan, 1992).And this is untrue!Regrettably, study sections have had too much power over what projects should proceed and which ones should be scrapped.As we have stated before, "such 'science by consensus' is unhealthy for the unfettered and productive pursuit of biomedical science" (Düzgünes ¸, 1999).
We challenge the Science Advisor to the President, and the leadership of NIH, NSF, and the U.S. National Academy of Sciences, and funding agencies throughout the world to refute each of the following 15 major problems with the current NIH and NSF grant systems.If they cannot, however, and we believe they cannot, we ask these institutions to implement more equitable alternatives that will not constrain the progress of science.

Problems with peer review
The NIH Peer Review document describes the mission of NIH to be seeking "fundamental knowledge about the nature and behavior of living systems and to apply that knowledge to enhance health, lengthen life, and reduce illness and disability."The document claims that the "NIH has a longstanding and time-tested system of peer review to identify the most promising biomedical research" (NIH Peer Review, 2019).During the initial peer review, the scientific merit of a grant application is evaluated by the Scientific Review Group that comprises scientists with relevant expertise in the area.The second review is the responsibility of the National Advisory Councils or Boards that decide on funding a proposal as well as on research priorities.Despite the claims of NIH that this is a longstanding and time-tested review process, it has never been compared scientifically to an alternative system, with respect to scientific productivity and breakthroughs, new therapeutic modalities, patents and its psychological, personal and scientific impact on grant applicants who do not "succeed." Furthermore, NIH has to process over 80,000 applications a year, utilizing over 25,000 reviewers (NIH Peer Review, 2019), an extremely wasteful system, possibly costing $1 billion/year, including the efforts of grant applicants and their post-doctoral fellows, graduate students and administrative assistants (Pagano, 2006).

REVISED Amendments from Version 2
In this second revised version of the article, we have addressed the concerns of the reviewers about the alternative funding system we have proposed.
We have indicated that scientists who opt into the alternative system can still apply for shared instrument, center or training grants.We have stressed that competition for funds in the traditional system would be drastically reduced because there would be 7-fold fewer grant applicants who would have opted for the new system.We have tried to allay the concern of one reviewer over the highly unlikely unintended consequences of the new system of funding.
Any further responses from the reviewers can be found at the end of the article Here we expound the major shortcomings and problems of peer review as it applies to the evaluation of grant applications.Some major breakthroughs in biomedical sciences have not been funded by NIH or NSF.There have been several publicized cases of highly important research not being given grant support that have later gone on to be recognized as significant scientific discoveries.Nobel Prize winner Stan Prusiner was not able to obtain NIH funding for studying prions early on in his research (Düzgünes ¸, 1998).Craig Venter's proposal to apply his whole-genome sequencing method to sequence a bacterial genome was not funded by NIH, and Nobel Prize winner Leon Cooper's work on neural networks was not supported by either the NIH or NSF (Bendiscioli, 2019).The most recent example is the work of Katalin Karikó, the winner of the Nobel Prize in Physiology or Medicine in 2023, who could not obtain funding for her groundbreaking work while at the University of Pennsylvania (Mueller & Kolata, 2023).These examples should have been a history lesson for funding organizations like NIH and NSF, which we pointed out 25 years ago (Düzgünes ¸, 1998)!Grant reviewers are competitors of applicants.If they are truly "peers", grant review panel members are very likely to be competitors of the grant applicant, even if not directly on the subject of the proposal.Thus, they will not be inclined to give the benefit of the doubt to an innovative research proposal that has not already been substantially carried out, particularly when they are struggling to procure funding themselves.
Discoveries are made before grant awards.The requirement and expectations for preliminary data in most grant applications indicates that a scientific discovery is expected to have already been made.Thus, the NIH and NSF may not be funding discoveries, but merely funding "mopping up operations," in the words of Thomas Kuhn (1962), unless the preliminary data have been generated by a previous grant, which may have been on an entirely different subject.
Reviewer critiques may be inaccurate, but without the responsibility and accountability of making inaccurate statements.Reviewers appear to have a mission to severely criticize applications to be able to weed them out, usually without the requirement to provide a published reference for any criticism.The reviewers are never accountable for their false statements or their scores (Swift, 1996), even though they can derail scientific careers and the advancement of a field of science.
Criticism never ends.Grant applicants may re-apply after revising their proposal to respond to the written critique of the review panel.However, the panel may have new members at this later time and may then have entirely new criticisms.In essence, if the review panel does not want to fund an application, it will find ways not to fund the application, revealing the whims of the individual reviewers.
Early career reviewers trained in a narrow area of science often think that valid science is what they are trained in.Thus, they may prevent the progress of science that may otherwise produce significant insights or therapeutic approaches to treat diseases.This problem was emphasized by Costello (2010): "… the new generation of grant reviewers judge grant proposals through the myopic lenses of their specialties ….Important ideas and proposals that lie outside the current interest in molecular biology are unlikely to get a credible and knowledgeable review …" Nonscientific, unpublished review criteria.Reviewers tend to use nonscientific criteria when making funding decisions.These include: (i) "probability of success" which would favor projects proposing only incremental advances and no risktaking; (ii) "level of enthusiasm" which is highly subjective and depends on the reviewer's mood at the time; and (iii) "grantsmanship" which is essentially rendering grant-writing a game, expecting particular approaches to the project.Implying the nonscientific nature of the evaluation process, the study by Pier et al., (2018) has shown that there is very little agreement between reviewers evaluating the same NIH grant applications.
Translational projects may require long-term funding.Projects that need additional time and experimentation to translate basic findings and initial discoveries into therapeutics or diagnostics may be considered by reviewers not to be innovative, thereby precluding the rapid development of a product that could diagnose or treat diseases.An example of this problem with NIH peer review is our inability to obtain grant funding since the mid-2000s for our research to develop gene therapy for oral cancer based on our initial discoveries (Neves et al., 2009), despite many applications.
Robbing Peter to pay Paul.Principal investigators may need to channel the funds of an existing grant to produce preliminary data for a new application in a new research area, instead of performing the funded experiments.Thus, experiments described in detail in applications may never be carried out and may essentially have been written only to convince reviewers to fund the grant application.In our view, this practice is unethical.It also demonstrates the absurdity of requiring preliminary data.
Precious scientist time is wasted on grant applications.Investigators spend a large proportion of their time on grant applications, which necessarily takes them away from their currently funded projects, if they are indeed grant recipients.This is not only counter-productive but may also be time paid by salary support from the granting agency, time that should have been spent on the funded project.A study by Kulage et al. (2015) calculated the cost of preparing a grant application.Principal investigators in this particular field spent between 70 to 162 hours per grant, and research administrators spent 34 to 66 hours, at a cost of USD $4,784 to $13,512.They estimated that, because funding rates are in the range 5-15%, a grant that is eventually funded would cost $72,460-$270,240.They concluded that "less costly and more efficient models of research funding are needed for the sustainability of the nursing profession" (Kulage et al., 2015).Scientists who have spent years in training and in research should be spending their time on scientific research and not on bureaucracy.
Describing experiments to be performed in five years is unrealistic.The elaborate description of experiments that will be performed three or five years in the future in a grant application contradicts the true nature of scientific research."Indeed, if the scientific enterprise were predictable, science would be banal, perhaps even boring" (Pagano, 2006).Thus, for reviewers to expect meticulous descriptions, as if this is how science advances, goes against the true nature of science.Science is driven by the insights of scientists and new discoveries, and often requires immediate changes in approach or direction.
Waiting for grant funding hinders scientific progress.Many fields advance rapidly while investigators are waiting for their grant applications to be evaluated and funded.If the investigator is not funded independently, the project barely moves forward.With the uncertainty in grant funding, we cannot afford science to progress at this slow and saltatory rate.
The human and material cost of NIH peer review.The administration of approximately 80,000 applications and 25,000 reviewers per year (NIH Peer Review, 2019) costs NIH and the research community both money and time that could have been used for actual research.Pagano (2006) has estimated that the cost of the preparation of grant applications in terms of the salaries of the applicants, their post-doctoral fellows, students and administrative assistants, and the time spent by all the reviewers could be more than $1 billion/year.For reviewers, evaluating grant applications is a chore performed for the sake of recognition and prestige, and perhaps to increase their own chances of obtaining funding.This can result in the compromise of objectivity by the reviewers, and even resentment, because of the inordinate amount of time required to complete a review.NIH officials conducting sessions at scientific meetings on how to write grant applications admit that reviewers may not be able to spend quality time on reviewing applications.Of course, this is never admitted in print, since peer review is supposed to be unquestionably the ideal system for funding science.
NIH scientists do not compete for grant funding.Although NIH provides extramural funds following grueling peer review of grant applications, its own scientists do not have to compete for this type of funding.Thus, NIH itself appears to have recognized the extreme drawbacks of the peer review system, enabling its intramural community to undertake longterm projects with stable funding and large laboratory groups.If peer review is such an indispensable system for funding science, why does NIH not implement this system for its own scientists?Why is the extramural scientific community considered second-rate citizens who must clamor for funding all their lives?Review panel scores do not predict success.An analysis of 102,740 funded grants has shown that percentile scores generated by NIH review panels for the applications are poor predictors of publication and citation productivity (Fang et al., 2016).Thus, the meticulous scoring process is essentially useless.Arturo Casadevall of Johns Hopkins University and the senior author of this study is quoted as saying "A negative word at the table can often swing the debate.And this is how we allocate research funding in this country" (Johns Hopkins Bloomberg School of Public Health, 2016).
"Interventions" to peer review An extensive study, published after the original version of the current paper was submitted, listed 38 "interventions" to peer review that would remedy its shortcomings (Kolarz et al., 2023).These interventions included limiting the number of applications from an institution, "dragon's den style pitch," interviews, moderation panels, applicants assessing proposals from other applicants, wildcard by each panel member to ensure funding of a particular project, sequential application of funding criteria, use of quotas, calling out unconscious biases, training "good reviewers," and open review and rebuttal.In this study, proponents of peer review appear to be grasping at straws to save peer review.
The frustration with the shortcomings of peer review has led some funding agencies, including the Health Research Council of New Zealand and the Volkswagen Foundation in Germany to revert to random selection of grantees by lottery (Avin, 2019).Schaubroeck (2022) has argued for a mixture of competitive and non-competitive funding.The system proposed in the next section is in line with this reasoning.
A simple and rational alternative to peer review We previously proposed a simple alternative to the current peer review system (Düzgünes ¸, 1999(Düzgünes ¸, , 2007)).This new system would provide continuous and stable funding for 10-year periods to scientists with a track record of solid publications (Düzgünes ¸, 1999) and to young scientists starting their first independent positions in a university or a research institute (Düzgünes ¸, 2007).Scientists opting for this mode of funding would merely submit a letter of intent with a one-page broad outline of their research direction.They could be chosen based on criteria including publications, citations and potential impact of their research field, by an international group of both established and young scientists who are not in a position to receive funding from NIH or NSF and are thus not competitors, would establish objective criteria for funding.These scientists would not act as peer reviewers of any grant applicant or application.They would merely set the objective criteria by which scientists applying for the new system are ranked, without knowing anything about the scientist, their field of research or the general topic of the research the scientist wishes to pursue.The criteria may include the number of citations and publications, the d-index (Di Caro et al., 2012), the h-index (Hirsch, 2005), and other factors the panel may consider to be useful.If such ranking is too rigorous and eliminates too many of the applicants (which could be calculated almost instantaneously), the panel would relax the criteria to include more of the applicants.Under this new system, NIH grants to scientists with the appropriate ranking would be limited to $400,000 per year of direct costs.These grants would be phased in over several years, up to 40,000 grantees.With indirect costs limited to 30%, all of these grants would cost $20.8 billion per year.Investigators who opt for this system would agree not to apply for the competitivie grants, so that they would not have an unfair advantage, although they could apply for shared instrument, center, or training grants.If they opt to obtain increased funding through the competitive system, they would have to forego their alternative grant (except for the last 1 year) and struggle like the rest of their competitors!Grants to young investigators would be set at $150,000 per year, with the same indirect cost rate.Forty thousand such grants would cost NIH $7.8 billion.Thus, at a total cost of $28.6 billion the NIH could fund 80,000 such grants, with minuscule expenses for scientific review.
Since individual applicants will not be identified during the establishment of the objective criteria, it will be impossible for there to be any discrimination towards the applicant on the basis of gender or minority status.
To accommodate the budget of the 80,000 grants, the number of competitive grants would be initially cut in half, to about 5,655 RPGs, at a cost of $12.2 billion.Since the proposed system will be phased in, and the NIH budget is likely to increase within the next few years, we anticipate that there will be no undue burden on the traditional grants and intramural funding at NIH, as the proposed system is implemented.In fact, competition for funds in the traditional system would be drastically reduced because there would be 7-fold fewer grant applicants who would have opted for the new system.
Possible unintended consequences of the alternative system Would there be unintended consequences of implementing the proposed alternative?It is theoretically possible, but extremely unlikely, that a scientist's track record is exaggerated by the availalble data, resulting in a grant award, and that they do not make any significant scientific progress during the 10 years of funding.Although this is highly unlikely, the loss of scientific progress in these circumstances should be compared to the loss of productivity by a laboratory that cannot obtain funding or cannot renew a grant, resulting in the departure from the laboratory of the trained personnel, or closing of the laboratory altogether.
We can also envision a scientist who opted for the novel funding mechanism, but did not make the cutoff of scientific measures used to establish the track record.First, there could be an appeal for a review of the scientist's track record.Even if this were to fail, the scientist still has the option to apply for the competitive grants.In fact, this is essentially the situation with the current peer review system, where about 80-90% of biomedical scientists find themselves.The frequency of instances of such unintended consequences, could be quantified by the granting agency.If these potentially very few instances are considered to be inefficiencies of the system we are proposing, we are inclined to ask if there is any efficiency in the current peer review system!We contend that, since the ranking or evaluation system is without reviewer bias (there being no reviewers), it is orders of magnitude more reliable than the "science by consensus" system.
Other alternatives to the peer review system It is instructive to note the findings of Azoulay et al. (2011) in comparing Howard Hughes Medical Institute awardees and NIH grant recipients.They reported that "selection into the HHMI investigator program-which rewards long-term success, encourages intellectual experimentation, and provides rich feedback to its appointees-leads to higher levels of breakthrough innovation, compared with NIH funding-which is characterized by short grant cycles, predefined deliverables, and unforgiving renewal policies.Moreover, the magnitudes of these effects are quite large."Scientists chosen for this program have a track record of scientific productivity, similar to our proposal, which, however, will have a broader base of scientists.
Vaesen and Katzav (2017) analyzed the proposal to "distribute available funds equally among all qualified researchers, with no interference from peer review."Their analysis indicated that "researchers could, on average, maintain current PhD student and Postdoc employment levels, and still have at their disposal a moderate (the U.K.) to considerable (the Netherlands, U.S.) budget for travel and equipment."Our proposal combines this equitable distribution of funds with the option for scientists undertaking very expensive projects to apply for the remaining highly competitive funds.
Evaluating the scientific success of grants obtained via peer review and the alternative system proposed here The paradigm shift we are proposing does not end here.The scientific productivity of scientists in these two categories over a 5-year and 10-year period will be analyzed, in terms of publications, citations, significant discoveries, patents, the d-index and h-index of the grant recipients, and the development of therapeutics, per dollar amount spent.These criteria would be weighted by the same or similar international panel that established the initial cutoff criteria for grantees in the new system.This evaluation would then prove or disprove the long-held assumption that peer review is the best system to award NIH or NSF grants.
If the current peer review system is shown not to be superior to the alternative we are proposing here, would it be abandoned?We would hope so.What, then, would become of the projects that necessicate larger budgets?We suggest some possibilities: Several investigators in the alternative system would pool their resources to tackle such projects.Centers focusing on a particular health condition that require very large budgets, such as autism and Alzheimer's disease centers, would be funded separately, possibly from the budget of the competitive grants.Investigators funded by this mechanisms, however, would not be eligible to be included in the alternative system we are proposing.
As we have indicated previously (Düzgünes ¸, 1999), "The United States has expended enormous capital in the training of its scientists.The scientific potential of the more than 80 percent of biomedical scientists who are unable to procure grants is too precious a resource to waste."

Conclusions
While contemplating writing this section, we came across an e-mail sent to potential NIH grant applicants and a separate website aimed at academics and including advice on grant applications as part of an industry aimed at grant applicants for "winning" reviews.The e-mail advertised that their program enabled the participants to "successfully write for reviewers".If an applicant is writing to impress a particular reviewer, the detailed norms, supposed objectivity, and scoring system of NIH peer review becomes questionable.Another website gave the advice to involve the reviewers' "reptilian brain" and went on to say that the written review of a grant application may come from the rational, cerebral layer of the brain, but the decision on whether the grant is awarded or not actually comes from the most instinctual layer.What has become of the best method to review grant applications?Considering all the problems of peer review of grant applications, we ask the Science Advisor to the President, and the leadership of NIH, NSF, the U.S. National Academy of Sciences and other funding agencies throughout the world to implement more equitable alternatives that will not constrain the progress of science.A staring point is the very simple and highly cost-effective alternative we have proposed here.

Jamie Shaw
Leibniz Universität Hannover, Toronto, Canada In this paper, Professor Düzgüneş offers a fairly thorough review of the criticisms against peer review in science funding policy and challenges administrators at various institutions to address these criticisms or else change directions.Professor Düzgüneş further offers a rough-and-ready proposal for an alternative funding scheme that leads to a more egalitarian funding distribution.I too am of the opinion that peer review is in a great deal of peril.The more it is used, the more we risk suffering from its problems and so the urgency of Professor Düzgüneş' challenge is entirely warranted.A response is also necessary, so that the NIH or NSF (or whoever) can contribute to the transparency of their decision making.I am, moreover, entirely sympathetic to more direct funding practices in the ballpark of those suggested by Professor Düzgüneş.Before providing some more critical remarks on this important paper, I'd like to highlight that I do not have Professor Fang's expertise with regards to the necessary funding levels for individual labs and withhold judgment about whether the dollar amounts could be made to work.This is an important objection that must be answered satisfactorily.In any case, such a budgetary analysis would require some contextual focus since expenses in theoretical projects are quite different from those in experimental contexts (and even within experimentalists, there are those starting a new lab and those who are able to borrow/use existing infrastructure).This provides wiggle room for a more nuanced distribution recommendation but, again, I leave this discussion open to others who know more than me on this subject.My only point here is that this debate will only be substantive focused on more specific institutional policies.I am in agreement with Professor Fang's remark about the presentation of the criticisms of peer review.The reason is that some of these objections, when isolated, are not genuine objections.For example, regarding the section under the heading "Some major breakthroughs in biomedical sciences have not been funded by NIH or NSF."This point on its own, while rhetorically powerful, is not particularly strong.Given the sheer number of applications, the NIH is bound to miss some breakthroughs.The real question is whether these examples are symptomatic of deeper, systematic patterns.They are, and this point is raised latter, but this underlying problem is the objection -not the mere fact that they missed out on breakthroughs.I suggest merging these objections, rather than separating them.Similarly, "Robbing Peter to pay Paul" seems more like a corollary to "Discoveries are made before grant awards" than a separate objection.
In other cases, the section header seems misleading."Criticism never ends", for example.The real worry here is that criticisms (also praise?) waver drastically amongst reviewers.The best up-todate meta-study on studies on this front that I am aware of is Guthrie S et.al, 2017 (ref [5]).In other cases, the objections are most likely legitimate but lack citations (e.g., "Grant reviewers are competitors of applicants", "Discoveries are made before grant awards").While these points are often discussed informally, analytic treatment of them that has been published should be incorporated here.Finally, the objection that "Translational projects may require long-term funding" at least needs to acknowledge that the NIH and NSF (and many other institutions) have dedicated special attention to translational research that can be extremely sizeable.The criticism here seems also to not concern peer review, more over, but the amount of funds available for translational research.The same is true for "Describing experiments to be performed in five years is unrealistic" which, if true, is not a problem of peer review per se but a problem of what is expected to be reviewed.Finally, it seems like a missed opportunity in the section "Nonscientific, unpublished review criteria" to not spend some time discussing the myriads of biases that discriminate on the basis of gender, age, ethnicity, etc.This is a massive objection against peer review and should be noted with some detail here.With regards to the positive proposal, I have strong reservations about the proposed criteria for assessment focused (seemingly exclusively) on publications and citation counts.This would likely lead to inadvertent discrimination.Many scientists have CV gaps that aren't a result of anything problematic (e.g., they had children, had to take care of their families, spent time doing public engagement instead of publishing, etc.).Not only should publications not be treated as a sine qua non of science and be used as a sole (or even primary basis) of funding, but 'productivity' measured in this way will marginalise already marginalized scientists who can be perfectly capable and talented members of the scientific community (see ref [3]; ref [2] ; ref [9] for starters).This will surely decrease the diversity of the scientific community as well which carries strong implications for innovation in science (Ginther DK. et.al, 2018 ref [4]) found that funding disparities amongst black and white applicants at the NIH were mostly caused by productivity gaps.Moreover, in the biomedical sciences in particular, there are worries that focus on productivity leads to more research funds going towards scientists with connections to for-profit companies which can bias their research (ref [6]; ref [7]).The same is true of citation counts, which are often not indicative of quality of work and stem from visibility issues or topic choices.Indeed, the authors motivation to improve innovation in science will likely be held back by promoting citation chasing, which often involves jumping on board with the latest trendy bandwagon.Moreover, and perhaps this is what Dr. Recio-Saucedo meant by "changing research culture", we are increasingly coming to recognize the value of many activities that are not 'productive' in the sense of generating publications.Increased attention to pedagogy, public engagement, advisory work, and so on all become further disincentivized and marginalized if we make funding dependent on publications alone.There are also some research programs that do not demonstrate their value in terms of publications.For instance, research on public health in Indigenous communities on the west coast of Canada requires decades to properly integrate researchers into the community, understand local norms and needs, etc. and most of this work is not publishable (see Castleden et al. 2015).For these reasons, I strongly suggest moving either towards even more minimalistic criteria or at least incorporating a broader sense of what counts as being 'productive.'Some smaller, but still important points include: "possibly costing $1 billion/year, including the efforts of grant applicants and their postdoctoral fellows, graduate students and administrative assistants (Pagano, 2006)."This number is a guesstimate from 2006.It is probably much more expensive 1.

1.
nowadays.A global study worth mentioning might be ref [1]."probability of success" is a scientific criterion, although the author is correct to suggest that this metric favor conservative research.

"Review panel scores do not predict success"
This is an oversimplification.In my meta-study, I saw some predictive success under the 5-year mark, but poor predictive success further into the future (ref [8]).That being said, I agree with Dr. Recio-Saucedo when they say that "Given the broad range of impacts that result from research, and the efforts on responsible use of metrics that are addressing known issues with research culture, the conclusion that a scoring system is not useful based on its ability to predict publication and citation productivity is to some extent reductive."The studies cited in this meta-study use citation counts as proxies and I agree that this is too narrow of a criterion.
Positive results were also found in ref [10].These findings do not rely on citation counts. 1. 1.

3.
"In this study, proponents of peer review appear to be grasping at straws to save peer review."This is too dismissive of these efforts.Unless reasons can be given that these initiatives are not likely to make a major difference, this point is mere speculation. 1.

4.
"science by consensus" This phrase might be misleading.The 'consensus' is amongst a very small group of 'peers' while some readers might associate 'consensus' with a broader community consensus.Perhaps 'science by local consensus' would be more appropriate. 1.

5.
I suggest a more recent reference than Racker (1979) for defending peer review as the best system there is.
Peer review has been evaluating scientifically against alternatives quite a bit.Much of the papers cited in this paper do just this.
I remain grateful for the opportunity to review and learn from this article, and believe is provides a valuable contribution to the growing chorus of discontent with peer review of grant applications.Urgent and potentially radical experimentation is necessary and I believe Professor Düzgüneş has taken admirable steps in this direction in this article as well as his earlier work.I very much hope that science funding institutions take notice.
the paper and the thorough reviews.The author presents an interesting alternative idea for allocating scarce research funds.The ideas discussed in this paper are not only relevant to US funding agencies.

My comments
(1) I agree with Reviewer 3 regarding the term "science by consensus".At least it does not apply to all funding programs that projects are only funded if there is a consensus among the reviewers.
(2) "The peer review process that evaluates proposals has been claimed arbitrarily to be the best there is."This is a strong claim.Indeed, some studies suggest that the grant review is arbitrary; other studies show at least very low to low agreement [1][2][3].
(3) "This consensus system, however, has never been evaluated scientifically against an alternative."This is a strong claim.In recent years, work has been done to analyse alternative allocation systems (see e.g.[4][5]).
(4) "The success rate for National Institutes of Health (NIH) grants is currently 20% (NIH Report, 2022), and the "payline" for research grant applications by experienced principal investigators is 11% (NIAID Funding News, 2023)."It would be helpful to add a definition of "payline".
(5) "Some major breakthroughs in biomedical sciences have not been funded by NIH or NSF".I agree with Reviewer 3, I don't think this is a strong argument.
(6) "Grant reviewers are competitors of applicants."This is a strong claim that should be backed up by literature.It does not seem to be a general argument (outside of NIH and NSF) since reviewers often have to be from abroad.
(8) "Nonscientific, unpublished review criteria.Reviewers tend to use nonscientific criteria when making funding decisions.These include: [...] ."This is a strong claim that should be backed up by literature.
(9) "Precious scientist time is wasted on grant applications."Instead of discussing a single study in detail, it might be better to summarize the findings of various studies.Our recent article discussed the economic costs of writing grant proposals [7].Please note that this is not a suggestion to cite our paper but perhaps a source of relevant literature.
(10) "Scientists opting for this mode of funding would merely submit a letter of intent with a one-page broad outline of their research direction."Writing a one-pager is not time-consuming, but if the content of this one-pager is not relevant to funding decisions, why not skip it and simplify the optin process?
(11) "It is theoretically possible, but extremely unlikely, that a scientist's track record is exaggerated by the available data, [...]."I agree with Reviewer 3: Why is that so unlikely?
The authors could suggest how potential risks could be analyzed in (smaller?)(if possibly controlled) experiments.A system change without such a prior analysis is somewhat unrealistic in my opinion.
(13) "We contend that, since the ranking or evaluation system is without reviewer bias (there being no reviewers), it is orders of magnitude more reliable than the "science by consensus" system."In this context, it is unclear to me what 'reliable' refers to.
(14) "Other alternatives to the peer review system."This section needs to be improved by adequately describing the literature on these alternatives.The author claims that the current system for distributing NIH and NSF funds has significant problems, including low success rates, potential biases, and inefficiencies.Despite claims of its effectiveness, this system has not been rigorously compared to alternatives.The author proposes to try out, on a large scale, an alternative system which provides long-term, stable funding based on scientific track records, without the need for detailed preliminary data or exhaustive peer review.This approach aims to reduce bureaucratic overhead and promote more equitable distribution of research funds.

References
I should mention here that already before reading this paper I was highly critical of distributing funding by peer review of projects.Hence, I am not in the best position to be a very critical reviewer.It should also be no surprise then that I am reasonably sympathetic to the proposal of the author (as I would be of any system that drastically cuts the costs of distributing funding).However, I have a few problems with how the argument for this system is made in this paper: -Most importantly, the paper almost entirely ignores the substantial empirical literature on the topic.Much of this literature would support the author's claims, but some would not.I think the paper would be a lot stronger if it was embedded in this literature, and relied on the best evidence we have.This is currently not the case, and many empirical claims about the current system are made with very little empirical backing.
-The paper does not give due credit to the current system and the people taking part in it.I am highly critical of this system, and am convinced that it should be changed and replaced by alternatives such as the one the author proposes, but I would be more charitable in describing the system, if only to enable productive debate with the proponents of the current system.The paper is written in a very opinionated and sometimes almost aggresive style, which will only hamper such debates.
-I think it would be advisable to test a new system, like the one proposed here, before investing 30 billion dollars into it.For example, funders could set up small scale experiments and monitor outcomes.This way, multiple alternatives could be compared.It would be good if the author could explain why they don't choose such an approach.
-The paper has very little discussion of other alternatives, such as various lotteries.I think it would be interesting to include such a discussion, as any alternative to the current system has to compete with these proposals to be tested.
-The main shortcoming of the proposal, in my eyes, is that there is very little justification for the use of the criteria that the author proposes (h-index and other bibliometric measures).I agree that peer review is not reliable and lacks validity, but I have similar doubts about these alternative metrics as well.Indeed, peer review scores often are partially based on these metrics, so any reliability they have should partially show in peer review as well.The paper would be stronger if the author could give a clearer justification of their choice of criteria.
In short, I am sympathetic to the author's proposal, but I think the argument could be made in a more rigorous, empirically substantiated and constructive manner.

Small points:
-Unclear what the 1 to 2% rate of the Gates foundation was based on.If it is pure speculation, it is better to omit it I think.
-I'm not sure 'science by consensus' is the best possible name for this system, as consensus is neither necessary nor sufficient to get funding.
-"Thus, the reputation of the current system is not based on systematic comparisons against other unbiased and more rational systems."While I agree, I don't think it is productive to claim the current system is less rational (and perhaps more biased) than alternatives.After all, you just claim that there has been no systematic comparison.Perhaps just omit 'unbiased and more rational'.
-" Regrettably, study sections have had too much power over what projects should proceed and which ones should be scrapped.As we have stated before, "such 'science by consensus' is unhealthy for the unfettered and productive pursuit of biomedical science" (Düzgüneş, 1999)."It would be good to give reasons, rather than just state this claim.The one example you give is not a very strong reason.
-"Furthermore, NIH has to process over 80,000 applications a year, utilizing over 25,000 reviewers (NIH Peer Review, 2019), an extremely wasteful system, possibly costing $1 billion/year, including the efforts of grant applicants and their post-doctoral fellows, graduate students and administrative assistants (Pagano, 2006)."Perhaps compare this to the budget that is being distributed and with estimates of the cost of review in other countries (I add a reference below).
-"Some major breakthroughs in biomedical sciences have not been funded by NIH or NSF".While this is interesting, I'm not sure it is a strong reason.Scientific success is indeed hard to predict, but even if not all success can be predicted by peer review it might still be that overall it does a good job.To be clear: I don't think it does, but I also don't think peer review should be able to identify every potential noble prize winner in order to be a viable system.
-"Grant reviewers are competitors of applicants."This is an empirical claim that is not trivially true: many reviewers are from other countries, and panel members have to follow conflict of interest rules (and perhaps cannot apply themselves?).It would be good to have some kind of substantiation of this problem, rather than making readers take your word for it.
-"Thus, the NIH and NSF may not be funding discoveries, but merely funding "mopping up operations," in the words of Thomas Kuhn (1962)".It is strange to cite Kuhn for this, as he did think that normal science is incredibly productive and essential to the success of science.There is much progress to be made within Kuhnian normal science, and I see no reason why NIH and NSF would not want to fund discoveries in normal science.If you really adhere to Kuhn's view on science, there are no reasons to hope that all science cosists in paradigm shifts.
-"Grant applicants may re-apply after revising their proposal to respond to the written critique of the review panel.However, the panel may have new members at this later time and may then have entirely new criticisms.In essence, if the review panel does not want to fund an application, it will find ways not to fund the application, revealing the whims of the individual reviewers."The reasoning in this paragraph was not clear to me.It would also be good to discuss existing data on the success rate of resubmitted applications.
-"Early career reviewers trained in a narrow area of science often think that valid science is what they are trained in...." The point that peer review is conservative is interesting, but does not seem related to the paragraph before that.Perhaps make this a separate point?-Nonscientific, unpublished review criteria: It is not clear to me what scientific review criteria would be, and why only scientific criteria are relevant.For example, potential long term societal relevance seems very important, but is not a scientific criterion.The discussion of interrater reliability is very brief, and only cites on of the very many studies on the subject.I recommend citing and reading the recent meta-analysis on the subject, and perhaps making this a separate point (see references).
- -The human and material cost of NIH peer review.and Precious scientist time is wasted on grant applications could be combined, as they are both about the costs of the system (as is another paragraph higher up).
-NIH scientists do not compete for grant funding.I do not see why a system could be good in some cases and not in others.In fact, I am convinced that we should use different distribution systems depending on different things we hope to achieve.
-Review panel scores do not predict success.There is a lot of debate about this, and the phrasing here is too strong to reflect what the literature shows.There are some studies that show that review scores are somewhat predictive.I recommend including more references, and definitely some that disagree with your claim.
-I might have miscounted, but I saw only 14 italicized problems.
-Interventions to peer review: there is a substantial literature on potential interventions to peer review.As far as I know, none of these interventions are particularly successful, but it would be good to discuss this literature a bit.
-A simple and rational alternative to peer review: I fail to see why this system is more rational than peer review (unless the 'rational' means 'cheaper').That is, I see no reason to think that the criteria that the author proposes are more reliable in predicting success than peer review is.If I remember correctly there is a study that shows that bibliometric measures are better predictors of scientific success than peer review scores, but if this is the argument that the author is relying on, they should mention it (see refs) -"It is theoretically possible, but extremely unlikely, that a scientist's track record is exaggerated by the availalble data": Why is this unlikely?There is evidence that scientists 'game the metrics' by salami-slicing publications and boosting their track records as much as possible.The system proposed here would encourage such behaviour, and I don't see why that would be extremely unlikely.
-"We contend that, since the ranking or evaluation system is without reviewer bias (there being no reviewers), it is orders of magnitude more reliable than the "science by consensus" system."I don't see why this would be the case.Even if there would be no reviewer bias, we'd need independent reasons to think that the new decision criteria are reliable.
National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, Southampton, UK I acknowledge the author's thorough response to my previous review.
The paper undeniably proposes an interesting alternative to the current funding system.However, some key aspects of this alternative require further clarification before considering it fully viable.I'd like to note the following points that could be elaborated in future revisions of the alternative proposed: Early Career Researchers (ECRs): A publication-based ranking system might disadvantage ECRs due to their limited publication history.How would the new funding model address this challenge?
Unintended consequences: Researchers with career breaks or those who don't follow traditional research paths may be disadvantaged.Gaps in publication track record or limited number of publications are common in these groups.What measures would be put in place to counter these potential biases in the alternative model?
Research culture: Initiatives like DORA promote assessment of broader research contributions which look beyond publications (e.g., teaching, mentoring, data sharing).How does this alternative system acknowledge and support non-publication outputs of technicians, data scientists and others who make important contributions to research through other types of outputs (e.g., research software, datasets, economic models)?
Alternatives to peer review: The alternatives described in the literature aim to move beyond traditional peer review (this open peer review is part of these efforts).The collective knowledge of the scientific community plays a role in the research ecosystem and it's too valuable to not utilize.It is not necessarily 'grasping at straws to save peer review'.It is merely learning from current systems to try to design the best alternative to fund research whilst acknowledging the limitations of human enterprise.

Reviewer effort:
The proposed model would require uncompensated review work from international panels, who are ineligible to apply for funds under the alternative model.In addition, the mention of revising ranking criteria (potentially relaxing them) suggests the need for reviewer meetings across time zones to make funding decisions.The design of funding panels in this way raises concerns about volunteer burden.Furthermore, reviewer contributions, whether they involve two hours or two days must be recognized, and I do not necessarily agree with qualifying this work as minuscule.
Training: Familiarity with metrics like d-index and h-index might require training for reviewers.
Evidence-based approach: Addressing the limitations of peer review is commendable.However, determining the best alternative requires objective evidence comparing outcomes such as effort, fairness, cost and outputs/impacts of the research funded.This would guide the design of a system that truly benefits the research community, fosters scientific progress and supports delivery of societal benefits from research.An RCT instead of a 5-10 year evaluation as proposed might provide evidence in a shorter period of time.The author reviewed the manuscript based on some of the recommendations made in the original submission and that is appreciated.However, the questions raised in the first review were answered only partially and concerns about the alternative proposed remain.

Is the topic of the opinion article discussed accurately in
The section on unintended consequences seems to indicate that the misidentification of track records for some applicants would be, albeit unlikely according to the author, inevitable.This because instead of offering a mechanism to avoid such consequence, the author offers a comparison of this consequence to what already occurs with the current funding allocation system.In addition, the solution proposed for the other unintended consequence discussed, this is, applicants not making the cut off of scientific measures to establish a track record, is for applicants to resort to the current funding system, which the author strongly criticises.The purpose of identifying unintended consequences should be avoidance, otherwise, why change?Since it is not possible to predict the incidence of these unintended consequences, it is necessary to design solutions for the alternative proposed and not operate on the premise that, because it is unlikely to happen, then there is no reason to plan for a way to address the issue should it arise.
The alternative proposed relies heavily on researchers' track record.The impact that the alternative has on early career researchers and other groups that have been identified to be less likely to receive funds for research (and as a result have less publications which lead to the other criteria proposed by the author, e.g., citations, h-index) has not been addressed in the alternative proposed.Establishing the criteria without identifying individual applicants does not solve the issue that some groups have been historically disadvantaged and are not going to meet the cut off criteria being suggested.
The lottery mechanisms for funding allocation which have been tried or implemented still make use of peer review to identify applications that meet the criteria of a funding call.Random allocation or lottery mechanisms address other issues related to funding decisions, but do not remove the peer review process entirely.
I agree with the author that systems to fund research must be revised constantly to ensure transparency, efficiency and fairness in how decisions are made, and that there are opportunities to improve the current funding systems.However, the alternative proposed does not clarify the potential for research waste from duplication or poor scientific productivity.
A study to compare the current funding system with alternatives (e.g., sandpits, random allocation, distribution of research funds based on track record, etc.) would be highly valuable.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have
As mentioned in my earlier review, a 2006 NIH analysis found that measures of citation productivity peaked at mean funding levels of $750,000 per year.Thus, funding labs at a fixed level of $400,000 per year would reduce productivity for many labs, and this is not even accounting for the substantial effect of inflation since that time.The author criticizes this analysis because he argues that productivity should be measured relative to funding levels, but the fact remains that absolute productivity for many individual labs would decline under his scheme.Moreover, there is considerable variation in productivity among laboratories, and the impact of funding restriction would be greatest for the most highly productive labs.A study of more than 15 million researchers from 1996-2011 found that fewer than 1% of labs published at least one article per year throughout the study period, yet this small fraction accounted for 41.7% of all papers and 87.1% of all papers with more than 1,000 citations Ioannidis et al (2014 1 ).By providing a minimal funding level for all labs, the author's alternative scheme would potentially cripple the most productive labs.The author may argue that labs desiring higher levels of funding could opt out of the alternative scheme, but intensive competition among the most highly productive labs for a reduced portion of the funding pie would only intensify the unhealthy level of competition that we see today.
The author cites the HHMI investigator program as an example of a successful alternative system that leads to higher levels of breakthrough intervention.However, he fails to note that HHMI investigators currently receive more than $1 million in direct support each year and represent an elite group of scientists with established high levels of productivity.
Lastly, I would point out that $150,000 per year would be woefully inadequate support for young investigators and place them at a significant disadvantage compared to more experienced investigators receiving $400,000 per year.This is reminiscent of the NIH R29 FIRST award program initiated in 1986 to provide modest levels of funding to early-stage investigators.This program was discontinued in 1997 when it was recognized that R29 recipients were less successful in obtaining subsequent funding compared to those who initially received a standard R01 for quite some time: Is peer review the best mechanism (e.g., in terms of efficiency, fairness, ability to detect ground-breaking ideas, resistance to bias) to make research funding decisions?Given the worldwide use of this mechanism for a number of activities within the research landscape (research funding, academic promotions, publications), peer review has been extensively studied many times with the aim of finding alternative mechanisms to fund research, some of them having been more or less successful than others.
In the article the author describes an alternative mechanism distribute funds which would be relevant to US funding agencies.The potential impact of an efficient alternative to the use of peer review in funding decisions to the research practice is significant.Some of the key points in the article indicate: The author summarises peer review in research funding as 'cumbersome, expensive, and counterproductive'

○
The author presents 15 issues with the current peer review system (science by consensus), including: missing major scientific breakthroughs, unfair/unbiased process driven by competition from researchers working in fields similar/the same to that of the applicant, requirements for preliminary data, lack of accountability for reviewers, poor understanding of the value of trans-disciplinary proposals, poor reviewer agreement, lack of long-term funding, and others.

○
The author proposes the alternative to be implemented with a view to evaluate its effect compared to the current peer review system in 5-10 years, also raising a challenge to senior management of major US research funders and leaders.

Overall strengths
The article is well structured, interesting, and contributes to the literature on how to increase research efficiencies, decision-making transparency, and impact, which keeps large funding organisations across the world preoccupied and involved in constant efforts for improvement.Reading this led me to think about how I am indeed putting peer review into practice by writing my reflection on this article.At the same time I couldn't avoid thinking that for all its challenges, peer review is a mechanism that encourages conversation, debate, and advances in scholarship, and how open peer review offers avenues to address some of the challenges associated with peer review discussed by the author.

Areas for improvement
The literature used to illustrate some of the research in peer review is not particularly the most recent.Comprehensive analysis of alternatives to peer review, which covers issues associated to peer review at all stages of funding processes, are available (see for example Peter Kolarz, Anete Vingre, Aaron Vinnik, Antonio Neto, Carlos Vergara, Claudia Obando, Rodriguez, Kalle Nielsen, Laura Sutinen, Review of peer review.Online report.June 2023, https://www.ukri.org/wpcontent/uploads/2023/07/UKRI-060723-Review-of-peer-review-Final-report-revs-v2.pdf).Alternatives that substitute peer review at the funding decision stage have been published (e.g., the use of random allocation mechanisms to allocate research funds: Avin S: Mavericks and lotteries.Stud Hist Philos Sci.2019; 76: 13-23) as well as researchers' views of some of these approaches (see for example: Liu M, Choy V, Clarke P, Barnett A, et al.: The acceptability of using a lottery to allocate research funding: a survey of applicants.Res Integr Peer Rev. 2020; 5: 3).
The description of the alternative is summarised since it refers to previous publications, but the summary misses the key challenges and unintended consequences of implementing the proposed alternative.Identification of unintended consequences will allow for the development of relevant indicators that would generate data/evidence to evaluate the efficiency of the alternative in comparison to the traditional use of peer review.In addition, alternative proposed does not seem to offer a new way of allocation research funds, as it includes the use of peer review in the way that is currently used (i.e., a panel of peers selects applications based on given criteria).
I appreciate that the author is not bringing in the positives of peer review to this article and that this strategy adds emphasis to the argument presented, however the narrative highlights the inefficiencies and issues surrounding peer review without acknowledging that no system is likely to be a one-size-fits-all solution to all the challenges surrounding allocation of research funds.
Recognising unintended consequences of alternatives is key and recognising the pros of peer review would actually support the alternative proposed (since it utilises peer review).
In terms of the alternative proposed, there are some points that would benefit from clarification: The author writes: "Review panel scores do not predict success.An analysis of 102,740 funded grants has shown that percentile scores generated by NIH review panels for the applications are poor predictors of publication and citation productivity (Fang et. al., 2016).Thus, the meticulous scoring process is essentially useless." Given the broad range of impacts that result from research, and the efforts on responsible use of metrics that are addressing known issues with research culture, the conclusion that a scoring system is not useful based on its ability to predict publication and citation productivity is to some extent reductive.I understand that the author is presenting more issues around peer review, but the emphasis on publication and citation is running counter intuitively to how research culture is evolving.
The proposed alternative reads: "They [applicants] could be chosen based on criteria including publications, citations and potential impact of their research field, by an international group of both established and young scientists who are not in a position to receive funding from NIH or NSF and are thus not competitors" The alternative offered here is a variation of peer review, but does not offer a different method to assess funding applications that decreases potential bias in decisions or increase inter-rater agreement of scorers.As written earlier, some aspects of the alternative are not different from the current way of selecting applications based on given criteria.The author suggests that funding decisions be made by "established and young scientists who are not in a position to receive funding".Finding established and young scientists who cannot receive funding by the selected organisations to conduct the review and/or make funding decisions decreases the pool of potential reviewers.Thus the workload on eligible individuals can become unmanageable very rapidly.Another aspect is training those young scientists in assessing applications for awards/grants that they do not know in depth.There seems to be underlying risks to the proposed alternative that run the risk of affecting Early Career Researchers (ECRs) disproportionately.In addition, the author suggests that funding panels consider potential impact of their research field.How can potential impact be objectively assessed by reviewers/funding panels?How would criteria that takes into account publications and citations be supportive of ECRs whose list of publications would not be competitive compared to senior researchers?Are ECRs inadvertently ineligible to the alternative?
"Thus, at a total cost of $28.6 billion the NIH could fund 80,000 such grants, with minuscule expenses for scientific review." If the process to identify reviewers and go through the peer review process for the half of the NIHR extramural funds remain in place, how will costs decrease to the point that they become negligible?
Overall, this is a good paper that will benefit from including additional information on key challenges of a large-scale implementation of an alternative to the de facto mechanism used in the distribution/allocation of research funds, which currently also regulates all research and scientific activity (from funding to publication; recruitment and career progression).The reviewer refers to "the key challenges and unintended consequences of implementing the proposed alternative;" but does not give any examples.Nevertheless, this is an important point.As an unintended consequence, we may think of cases where the track records of a few awardees were misidentified, and they do not generate any significant publications during the 10 years of funding.We would argue, however, that the loss of scientific progress in these circumstances should be compared to the loss of productivity by a laboratory when it cannot renew a 3-year grant and all the trained personnel have to leave the laboratory.

References
By contrast, we could envision a scientist who opted for the novel funding mechanism, but did not make the cutoff of scientific measures used to establish the track record.In this case, the scientist always has the option to apply for the competitive grants.This is the situation with the current peer review system in which about 80% of biomedical scientists find themselves.The frequency of such incidences could be quantified by the granting agency.Whether this analysis could be translated into the "efficiency of the alternate in comparison to the traditional use of peer review" begs the question, "is there any efficiency in the current peer review system?"Our answer would be "no."We have added a section in the revised paper discussing the possible, but in our view minimal, unintended consequences of our alternative method of funding.
"In addition, the alternative proposed does not seem to offer a new way of allocation research funds, as it includes the use of peer review in the way that is currently used (i.e., a panel of peers selects applications based on given criteria)." We stated in our paper that the alternative method of funding scientists would be based on the track record of solid publications of the scientists, a system which we had first proposed in 1999 (Duzgunes , 1999).In 2007, we added the mechanism to provide funding to young scientists starting their first independent positions in a university or a research institute (Duzgunes , 2007).The reviewer interprets the work of the "international group of both established and young scientists who are not in a position to receive funding from NIH or NSF and are thus not competitors" as a form of peer review, possibly because of our use of the word "chosen."We should have given more details of what determines a "track record of solid publications" We did not intend this group of scientists to be involved in the choice of the scientists to receive funding, but merely to establish the track record criteria.Examples of such criteria are citations, publications, the h-index and the d-index (Di Caro et al., 2012).The international panel of scientists would only be involved in establishing the overall weight of such criteria.We have modified the manuscript to clarify this aspect of our proposed system.
"…however the narrative highlights the inefficiencies and issues surrounding peer review without acknowledging that no system is likely to be a one-size-fits-all solution to all the challenges surrounding allocation of research funds.Recognising unintended consequences of alternatives is key and recognising the pros of peer review would actually support the alternative proposed (since it utilises peer review).
One of our main contentions here is that a system that has been acclaimed for decades as being the best method to allocate research funding, has not even been tested side-by-side against an alternative, unbiased, and more rational system.As we have delineated above, we have recognized the positive aspect of peer review by proposing that the unequivocally objective criteria for the new system would be discussed and established by an international panel of junior and established scientists.Further pros of peer review may be seen in the review of manuscripts submitted to journals, where errors in the research or in the analysis of data may be identified.Even so, the rejection of breakthrough articles by Karikó and Weissman on mRNA by reputable journals, as reported in the NY Times, is not a plus for the reputation of peer review even for journal articles.
In terms of the alternative proposed, there are some points that would benefit from clarification: The author writes: "Review panel scores do not predict success.An analysis of 102,740 funded grants has shown that percentile scores generated by NIH review panels for the applications are poor predictors of publication and citation productivity (Fang et. al., 2016).Thus, the meticulous scoring process is essentially useless." Given the broad range of impacts that result from research, and the efforts on responsible use of metrics that are addressing known issues with research culture, the conclusion that a scoring system is not useful based on its ability to predict publication and citation productivity is to some extent reductive.I understand that the author is presenting more issues around peer review, but the emphasis on publication and citation is running counter intuitively to how research culture is evolving.
We are merely stating the conclusions of Fang et al. ( 2016), which may be reductive, but they provide strong support for our contention that the supposedly objective scoring system of grant peer review (which, by the way, has not been tested against an alternative system) does not lead to success in terms of measurable quantities, such as publications and citations.The Reviewer has not explained what she means by the thesis that these metrics are "running counter intuitively to how research culture is evolving."Nevertheless, we believe this issue is outside the scope of our article.
The alternative offered here is a variation of peer review, but does not offer a different method to assess funding applications that decreases potential bias in decisions or increase inter-rater agreement of scorers.As written earlier, some aspects of the alternative are not different from the current way of selecting applications based on given criteria.
In the revised manuscript, we have clarified the involvement of the international panel of scientists.They do not act as peer reviewers of any grant applicant or application.They merely set the objective criteria by which scientists applying for the new system are ranked, without knowing anything about the scientist, their field of research or the general topic of the research the scientist wishes to pursue.
"Finding established and young scientists who cannot receive funding by the selected organisations to conduct the review and/or make funding decisions decreases the pool of potential reviewers." The majority of scientists live in countries outside of the United States.Although some non-U.S.scientists receive funding from NIH or NSF, their number is extremely limited, and they would not be permitted to participate in the international panel.
"Thus the workload on eligible individuals can become unmanageable very rapidly" As we have clarified above and in the revised manuscript, the work of the international panel is minuscule compared to that of the NIH study sections or NSF reviewers.The panel merely determines the criteria of acceptance, such as the percentage weight of the d-index, the number of citations, the number of publications, and other criteria (e.g. the h-index) they may consider to be useful.If such ranking is too rigorous and eliminates too many of the applicants (which would be calculated almost instantaneously), the panel would relax the criteria.
"Another aspect is training those young scientists in assessing applications for awards/grants that they do not know in depth." As we have explained above, no such training would be required of any scientist.Nevertheless, the members of the review panel would have to become familiar with the dindex and the h-index, and perhaps other criteria.
"There seems to be underlying risks to the proposed alternative that run the risk of affecting Early Career Researchers (ECRs) disproportionately." On the contrary, in the scheme we have proposed, early career researchers beyond their post-doctoral training would be given 5-year grants to establish their own researach programs, without spending most of their time applying for grants, and most likely failing.This would help young scientists in obtaining tenure and keeping their laboratories and trained personnel.By contrast, in the current system, the fierce competition with established scientists puts early career researchers at a perilous disadvantage, and force a majority of them to close their labs and move to other universities, research institutions or industry.
In addition, the author suggests that funding panels consider potential impact of their research field.How can potential impact be objectively assessed by reviewers/funding panels?How would criteria that takes into account publications and citations be supportive of ECRs whose list of publications would not be competitive compared to senior researchers?Are ECRs inadvertently ineligible to the alternative?
We agree that considering the potential impact of the research field would be difficult; thus, we have deleted this expectation of the international panel.Regarding the ECRs, we have addressed this concern in the previous point above.In contrast to the Reviewer's contention, ECRs are currently at an enormous disadvantage, because they are expected to compete with all the established investigators.In the system we are proposing, ECRs have the opportunity to set up their independent research with the help of the 5-year grant they will be given when they start their first independent position.Depending on the NIH and NSF budgets, the period and the amount of funding to the ECRs could be increased from those we are proposing here.
If the process to identify reviewers and go through the peer review process for the other half of the NIH extramural funds remain in place, how will costs decrease to the point that they become negligible?
The system we are proposing here would enable investigators whose research needs are much greater than the $400,000 per year, to compete for funds just like they have been doing for decades.The advantage of the split system we are proposing is that these investigators would not have to also compete with the 80,000 other applicants.We proposed the split system of funding with the expectation that the two segments could be compared at the end of a 5-or 10-year period, in terms of research productivity, breakthroughs and patents (Duzgunes, 1999).The burden of peer review in study sections at NIH would be lowered, because there would be 80,000 fewer applications.
Competing Interests: No competing interests were disclosed.The commentary lists what it describes as 15 "major problems with the peer review process."In place of the current system, the commentary suggests awarding funds at fixed levels based on level of applicant experience, a one-page summary of research plans, and publication record.This would reduce administrative costs and substantially increase the number of funded researchers.Few of the issues with grant peer review or the solutions proposed in this commentary are new 1,2,3 .Nevertheless, the topic remains timely, as low success rates for grant applications persist.
1.The 20% success rate for NIH grants is a misleading indicator of the current difficulty in obtaining research funding, as success rates count original and resubmitted grants that are received during the same fiscal year as a single submission and combine different types of grants that have variable success rates.A better indicator would be paylines, which are at 10-12% for R01 applications from established investigators at many institutes.Thus, the situation is even more dire than the commentary suggests.
2. There is some redundancy in the list of "major problems," which essentially boil down to: (a) peer review is biased and unreliable; (b) the current funding system discourages innovation and long-term projects; (c) too much time is wasted on seeking funding or peer review instead of on research.Another concern, which is not mentioned, is that insufficient levels of public funding for research and development can have a disproportionate impact on female and underrepresented minority researchers 4,5,6 .
3. The commentary fails to consider potential disadvantages of the proposed funding strategy.For example, dramatically lowering the bar for research funding might result in an increase in lower quality research, because many labs whose work was not previously found to be sufficiently meritorious to warrant support would now be funded.Some institutions might be tempted to expand the size of their faculties in order to exploit the system, which would increase pressure on the research budget.Half of the extramural budget would be retained for competitive applications involving projects that require larger budgets, but this would essentially intensify competition among larger laboratories, which include some of the most productive research groups.The $400,000 per year budget falls far short of the $750,000 direct funding level that was found to be associated with peak measures of productivity in a 2006 NIH study 7 , and research costs have only risen since that time, suggesting that most labs would be underfunded under the new scheme.
4. The comparison of recipients of support from the NIH and Howard Hughes Medical Institute is invalid, because HHMI investigators represent a more highly selected subset of scientists with demonstrated high levels of productivity.
5. The proposed funding approach shares some features with the approach proposed by Pagano 1 , which should be cited.
6.A single paragraph suggests performing a 5-and 10-year formal study to compare the productivity resulting from the current and alternative funding approaches but fails to consider the many methodological challenges in performing such a study.
7. The commentary repeatedly refers to "we," but only a single author is listed.
Although I share many of the frustrations expressed in this commentary, I have concerns about the practicality of the proposed solution.Nevertheless, a dialogue on possible ways to address the inadequate current levels of research funding and the shortcomings of grant peer review should continue.This commentary may help to further that dialogue.Reviewer Expertise: microbiology, molecular biology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
10-12% for R01 applications from established investigators at many institutes.Thus, the situation is even more dire than the commentary suggests.
We have modified the Introduction to indicate this important point.
2. There is some redundancy in the list of "major problems," which essentially boil down to: (a) peer review is biased and unreliable; (b) the current funding system discourages innovation and long-term projects; (c) too much time is wasted on seeking funding or peer review instead of on research.Another concern, which is not mentioned, is that insufficient levels of public funding for research and development can have a disproportionate impact on female and underrepresented minority researchers We agree with the reviewer's summary of the major problems with peer review; but we would prefer to retain the detailed exposé of peer review in our article.Regarding the disproportionate impact on female and minority resesarchers, we believe that our proposed system of grant allocation will alleviate the problem of recognition of the names of applicants by review panelists and the consequent bias against female and minority scientists.We have added a statement to this effect in the revised manuscript.
3. The commentary fails to consider potential disadvantages of the proposed funding strategy.For example, dramatically lowering the bar for research funding might result in an increase in lower quality research, because many labs whose work was not previously found to be sufficiently meritorious to warrant support would now be funded.Some institutions might be tempted to expand the size of their faculties in order to exploit the system, which would increase pressure on the research budget.Half of the extramural budget would be retained for competitive applications involving projects that require larger budgets, but this would essentially intensify competition among larger laboratories, which include some of the most productive research groups.The $400,000 per year budget falls far short of the $750,000 direct funding level that was found to be associated with peak measures of productivity in a 2006 NIH study 7 , and research costs have only risen since that time, suggesting that most labs would be underfunded under the new scheme.
We have added a section in the manuscript that discusses the unintended consequences of the proposed funding strategy, as also requested by Reviewer 2. Lowering the bar for research funding is highly unlikely to result in lower quality research, considering that a project that, for example, scores 10 points lower than the score of the funded grants is not expected to be of that much lower scientific quality.In support of this contention is the current inclination of some funding agencies to use a lottery to fund the grant applications in the top tiers.
The reviewer in essence admits that large laboratories obtain the large grants, leaving individual, creative scientists (who may not be good academic politicians), without funds.In the revised manuscript, we have cited the recent case of Nobel Laureate Dr. Katalin Karikó, who could not obtain funding during the period of her groundbreaking work on modified mRNA.The alternative system we are proposing would enable scientists like Dr. Karikó to do their work without having to become a junior member of a large lab.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com

Is the topic of
the opinion article discussed accurately in the context of the current literature?Partly Are all factual statements correct and adequately supported by citations?Partly Are arguments sufficiently supported by evidence from the published literature?Partly Are the conclusions drawn balanced and justified on the basis of the presented arguments?Partly Competing Interests: No competing interests were disclosed.Reviewer Expertise: Science of Science; Intelligent Systems I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have Stijn Conix UCLouvain, Louvain-la-Neuve, Belgium

2 Reviewer
the context of the current literature?Partly Are all factual statements correct and adequately supported by citations?Partly Are arguments sufficiently supported by evidence from the published literature?Partly Are the conclusions drawn balanced and justified on the basis of the presented arguments?Partly Competing Interests: No competing interests were disclosed.Reviewer Expertise: I am Senior Research Fellow at the National Institute for Health and Care Research within the Research on Research group and have conducted research on peer review.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Version Report 14 February 2024 https://doi.org/10.5256/f1000research.159084.r227840© 2024 Recio-Saucedo A. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Alejandra Recio-Saucedo National Institute for Health and Care Research (NIHR) Coordinating Centre, University of Southampton, Southampton, UK the context of the current literature?Partly Are all factual statements correct and adequately supported by citations?Partly Are arguments sufficiently supported by evidence from the published literature?Partly Are the conclusions drawn balanced and justified on the basis of the presented arguments?Partly Competing Interests: No competing interests were disclosed.Reviewer Expertise: I am Senior Research Fellow at the National Institute for Health and Care Research within the Research on Research group and have conducted research on peer review.

Reviewer Report 12
September 2022 https://doi.org/10.5256/f1000research.136253.r149008© 2022 Fang F. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Ferric Fang Department of Laboratory Medicine and Pathology, University of Washington, Seattle, WA, USA This commentary is a critique of the approaches used by the U.S. National Institutes of Health and National Science Foundation to allocate research funding.
Are arguments sufficiently supported by evidence from the published literature?YesAre the conclusions drawn balanced and justified on the basis of the presented arguments?PartlyCompeting Interests: No competing interests were disclosed.
Reference Source Kolarz P, Vingre A, Vinnik A, et al.: Review of peer review: final report.June 2023.(accessed October 16, 2023).Reference Source Kuhn TS: The Structure of Scientific Revolutions.Chicago: University of Chicago Press; 1962; 264 pp.Kulage KM, Schnall R, Hickey KT, et al.: Schaubroeck T: Research funding: dial back the competition.Nature.2022; 610: 257.PubMed Abstract|Publisher Full Text Swift M: Innovative research and NIH grant review.J. NIH Res.1996; 8(12): 18. Vaesen K, Katzav J: How much would each researcher receive if competitive government research funding were distributed equally among researchers?PLoS One.2017; 12(9): e0183967.PubMed Abstract|Publisher Full Text 1. Mayo NE, Brophy J, Goldberg MS, Klein MB, et al.: Peering at peer review revealed high degree of chance associated with funding of grant applications.J Clin Epidemiol.2006; 59 (8): 842-8 PubMed Abstract | Publisher Full Text 2. Mutz R, Bornmann L, Daniel HD: Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach.PLoS One.2012; 7 (10): e48509 PubMed Abstract | Publisher Full Text 3. Erosheva E, Martinková P, Lee C: When Zero May Not Be Zero: A Cautionary Note on the Use of Inter-Rater Reliability in Evaluating Grant Peer Review.Journal of the Royal Statistical Society Series A: Statistics in Society.2021; 184 (3): 904-919 Publisher Full Text 4. Shaw J: Peer review in funding-by-lottery: A systematic overview and expansion.Research Evaluation.2023; 32 (1): 86-100 Publisher Full Text 5. Reference Source 6. Herbert DL, Barnett AG, Clarke P, Graves N: On the time spent preparing grant proposals: an observational study of Australian researchers.BMJ Open.2013; 3 (5).PubMed Abstract | Publisher Full Text 7. Reference Source "An example of this problem with NIH peer review is our inability to obtain grant funding since the mid-2000s for our research to develop gene therapy for oral cancer based on our initial discoveries (Neves et al., 2009), despite many applications."Ithink the paper would be stronger if the arguments relied less on single examples.-Ina recent survey my co-authors and I did on QRPs in research funding, we had one question about this problem.It could be relevant to mention that result here.

the topic of the opinion article discussed accurately in the context of the current literature? Partly Are all factual statements correct and adequately supported by citations?
1. Kolarz P, Vingre A, Vinnik A, Neto A, et al.: Review of peer review.Online report.June 2023.2023.Reference Source 2. Bollen J, Crandall D, Junk D, Ding Y, et al.: From funding agencies to scientific agency: Collective allocation of science funding as an alternative to peer review.EMBO Rep. 2014; 15 (2): 131-3 PubMed Abstract | Publisher Full Text 3. Fox C, Albert A, Vines T: Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution.Yes development of relevant indicators that would generate data/evidence to evaluate the efficiency of the alternative in comparison to the traditional use of peer review."

Is the topic of the opinion article discussed accurately in the context of the current literature? Yes Are all factual statements correct and adequately supported by citations? Yes
5. Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, et al.: Sex differences in application, success, and funding rates for NIH extramural programs.Acad Med.2011; 86 (6): 759-67 PubMed Abstract | Publisher Full Text 6. Lauer M, Roychowdhury D: Inequalities in the distribution of National Institutes of Health research project grant funding.eLife.2021; 10.Publisher Full Text 7. Wadman M: Study says middle sized labs do best.Nature.2010; 468 (7322): 356-357 Publisher Full Text